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Kurl  V^anLehn 

Learning  Research  and  Develojniieiil  Ceiiler 
University  of  Pittsburgh 
PiUsbiirgh.  PA  ]52()0 


September  11,  19fl2 


Dr,  Susan  Chipman 
Scii'iitilic  Officer  Code  114‘2CS 
Office  of  Naval  Research 
800  N.  Quincy  St. 

Arlington.  VA  22217-5000 

Dear  Susan; 


This  letter  is  a  quarterly  progress  rejjort  for  the  On-Line  Assessment  of  E.xirertise  project 
(Crant  number  N00014-91-J-1 520).  It  covers  the  period  April,  1002  to  June.  10!)2. 

The  two  major  accomplishments  during  the  report  period  wore  a)  we  conducted  three  studies 
to  further  test  the  user  interfaces  and  b)  developed  an  initial  integrated  version  of  OLAE. 


The  first  stiuly  conducted  was  designed  to  test  two  user  interfaces  for  gathering  data  about 
a  stinh'iit's  qualitative  problem  solving  knowledge.  Ten  subjects  were  drawn  from  the  Pitt 
introductory  physics  class  for  science  and  engineering  majors  and  were  each  sfiowji  the  two 
interfaces.  Each  subject  was  first  introduced  to  the  qualitative  problem  solving  interface  and 
taught  how  to  use  it.  They  were  presented  with  20  qualitative  problems  to  solve.  Some  of  the 
problems  required  multiple  choice  answers  and  some  required  drawing  free  body  diagrams. 
All  of  th<’  problems  were  relevant  to  the  target  chapter  of  their  textbook  (Chapter  5 
Newton's  laws  applied  to  translational  motion)  but  were  .selected  from  different  texts.  After 
they  finished  answering  all  problems,  they  were  asked  to  orally  justify  each  answer.  This 
first  task  lasted  bet\\een  30  minutes  and  an  hour,  depending  on  the  subject. 

Alter  a  short  break,  they  were  then  introduced  to  the  difficulty  estimation  and  basic  ajiproach 
int<'rlac(’  and  taught  how  to  use  it.  They  were  presented  with  11  quantitative  problems  and 
for  ('ach  were  asked  to  estimate  difficulty  and  to  identify  a  basic  a])proach.  Specifically,  they 
w<»re  asked  to  specify  what  factors  of  the  juoblem  made  it  difficult  or  easy,  to  indicate  the 
probh'in'..  overall  difficulty  on  a  five-point  scale,  and  to  indicate  what  steps  they  would  take' 
to  generate  a  solution.  Again,  all  of  the  problems  were  relevant  to  the  target  chapter,  but 
weie  chosen  from  other  texts.  As  expected,  the  first  five  subjects  heljied  us  identify  se\<'iat 
significant  problems  with  the  two  interfaces.  Subsequent  analysis  will  be  based  oji  the  fi\e 
remaining  subjects. 

Th(’  second  study  that  we  conducted  was  designed  to  test  the  user  interfaces  for  example 
studying  and  problem  .solving.  We  have  already  e.xposed  .students  to  both  interfaces,  but 
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llic  siil)j<‘<  ts  in  our  oarlicr  study  wore  already  familiar  will)  the  malerial  in  ('liai)ler  5.  I'oi 
the  new  study,  we  wanted  to  lost  how  students  react  to  (he  interfaces  when  (Ik-  mateiial  in 
the  examples  and  problems  is  novel.  Five  subjects  were  drawn  from  (he  Pitt  introductoiy 
physics  class.  Each  suljjcct  participated  in  a  'i  hour  .se.s.sion.  First,  each  subject  was  given 
a  12  (piostion  jrretest  that  tests  knowledge  of  |)revious  chai)ters,  the  necessary  mathematics, 
and  r«'lovaut  conceptual  knowledge.  Each  was  then  introduced  to  the  system  and  taught  how 
to  use  (ho  poor  man's  ejetracker  and  the  editors  for  problem  solving.  They  were  iiext  gi\en 
a  i)ho(oco]>y  of  Chapter  5  with  the  worked  out  examples  replaced  by  white  s])ace.  I'hey  were 
instructed  to  read  the  chapter  and  whenever  they  encountered  a  white  s])ace,  they  were  to 
read  through  the  next  worked  out  example  using  OLAE's  i>oor  man's  eyet  racla-r.  They  wc-re 
also  aske<l  to  give  a  verbal  protocol  as  they  studied  the  exami>les  and  were  jrrompted  wIk-u 
they  fell  silent. 

When  the  subjects  finished  the  chapter  and  examjrles,  they  were  given  a  l)reak  and  then 
asked  (d  solve  a  set  of  quantiiativc  physics  piobIem.s.  They  worked  until  th<-  end  of  the 
d  hour  session,  solving  as  many  problems  as  they  could.  Again,  they  were  asked  to  give 
a  verbal  protocol.  Initial  analysis  of  the  protocols  and  problem  solving  ability  has  shown 
that  subjc'cts  who  produce  self-explaining  behavior  solved  more  of  the  problems  correctly. 
In  our  subject  pool,  self-explainer  also  performed  Iretter  on  the  jjretest.  Furthermore  .self¬ 
explainers  sjjent  more  time  reading  example  lines  on  average  and  had  a  larger  variance  than 
did  non-self-exj)lainers.  That  is.  the  non-self-explainers  s])ent  about  the  same  amount  of 
time  on  all  lines  while  the  self  explainers  spent  much  longer  on  some  lines  than  on  others. 

The  third  study  was  designed  to  test  the  user  interface  for  the  ])roblem  classification  task. 
Four  subjects  were  drawn  from  the  Pitt  introductory  physics  class.  Each  subject  was  first 
introduced  to  the  system  and  taught  how  to  classify  problems  into  categories  and  (aught 
how  to  name  the  categories.  They  were  then  pre.sented  with  30  problems  to  sort.  All  of  (he 
})roblems  were  relevant  to  the  target  chapter  but  were  selected  from  different  texts.  .After 
they  finished  .sorting  all  problems,  they  were  asked  to  orally  describe  each  category.  All 
four  subjects  uncovered  some  significant  problems  with  the  user  interface.  W'e  will  draw 
additional  subjects  to  test  the  repaired  interface. 

In  (his  re])ort  period,  we  have  also  completed  an  initial  version  of  OLAE,  that  integrates  (he 
four  major  components:  the  interfaces,  a  cognitive  domain  mode!  (CASCADE),  the  bayesian 
infereiKc  engine,  ami  an  assessor  s  interface.  For  this  initial  version,  we  isolated  the  subset 
of  tin-  (ognitive  domain  theory  that  models  how  ])eo])le  solve  kirn-mat ics  probh-ms.  Then 
for  each  problem  in  a  problem  set.  OLAE  generates  a  belief  net.  Each  resulting  Ix-lit-f  net 
cont  ains  nodes  for  every  action  that  a  studejit  might  ])eiform  oji  the  interface  and  nod<-s  for 
every  rule  in  the  cognitive  domain  theory  that  a  stmleiit  might  know.  The  belief  net  al.s(» 
has  paths  that  rei>iesent  how  the  rules  might  be  used  to  ])roduce  actions. 

When  a  student  is  then  ])resented  with  a  jvroblem  on  the  problem  solving  interface,  tin- 
student's  actions  are  collected.  The  nodes  in  the  ivelief  net  that  corresivond  to  actions  by 
(Ik-  stmlent  are  given  a  jvrobability  of  1.  The  belief  net  inference  algorithm  is  then  aivjvlied 
to  infei  which  rules  the  student  knows  or  does  not  know.  Finallv.  the  student  model  <an  be 
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vipwrd  witli  a  riKliiiieiilarv  assessor's  interface. 

Tlie  initial  integrated  version  of  OLAE  has  highlighted  some  difficulties  witli  our  curnuil 
apjjroacJi.  As  we  e.xpected,  the  generation  of  complete  belief  net  s  for  each  pioblem  is  far  too 
e.\pensi\e  and  will  become  worse  when  we  adoj)t  a  more  extensive  cognitive  domain  mode]. 
In  future  versions  of  OLAE,  we  will  use  an  incremental  method  for  building  belief  nets  that 
oidy  builds  the  portion  of  the  belief  net  that  is  necessary  to  analyze  a  subject's  actions.  A 
second  difliculty  with  this  version  is  that  because  students  are  able  to  ])erform  a  v<>ry,  very 
large  number  of  actions,  the  belief  nets  can  have  a  very  large  number  of  nodes.  This  is  true 
even  when  the  belief  nets  are  built  incrementally.  For  exam])Ie.  students  may  writ  e  etpiatioiis 
in  man\  e(|uivalent  wavs  and  may  derive  results  in  many  different  orders.  We  will  j)artially 
addr(',ss  this  problem  by  translating  tl)e  raw  actiojjs  froj))  the  interface  into  a  simpler  lormat 
for  t  he  Ix'lief  net. 


Sincerely. 

}L^ 

Kurt  A.  VajiLelm 

Associate  Professor  of  Com]>uter 

Science.  Senior  Scientist.  LRDC 
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ONH  Resident  Rejjresentative  NGG005,  OSU  Research  Center 
Director,  Naval  Research  Laboratory,  Code  2G27 
Defense  Technical  Information  Center 


Fniversity  of  Pittsburgh  ('ontracts  Office 
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